10 scraping reliability improvements for source ingestion pipeline - #81
10 scraping reliability improvements for source ingestion pipeline#81potemkin666 with Copilot wants to merge 3 commits into
Conversation
Agent-Logs-Url: https://github.com/potemkin666/Brialert/sessions/4a7abf7c-ad21-4b5a-a83c-fc5c5f2be1d5 Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
1. Expand Playwright browser fallback allowlist (7 more CT/police sources, budget 2→6) 2. Add RSS/Atom feed discovery as fallback for failing HTML sources 3. Add replacement endpoint / mirror fallback logic on 404/410/blocked 4. Diversify user-agents (4→12, Firefox/Safari/Edge, randomised retries) 5. Add proxy/IP rotation support (BRIALERT_PROXY_URL, BRIALERT_PROXY_LIST) 6. Implement adaptive timeout and retry tuning per source 7. Add Google News aggregator RSS feeds as redundant context sources 8. Improve HTML selector resilience with LD+JSON, OG meta, heuristic fallback 9. Implement smarter scheduling (least-recently-successful-first, higher RSS concurrency) 10. Add source health auto-recovery (shorter quarantine recheck, redirect persistence, URL discovery) Agent-Logs-Url: https://github.com/potemkin666/Brialert/sessions/a6d74a6e-c94a-433a-99ec-d50bd2605258 Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
… feed link matching, valid region for aggregator sources Agent-Logs-Url: https://github.com/potemkin666/Brialert/sessions/a6d74a6e-c94a-433a-99ec-d50bd2605258 Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
|
@copilot This branch has conflicts that must be resolved Use the command line to resolve conflicts before continuing. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 610968f6a3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const response = await fetch(url, { | ||
| headers: { | ||
| ...mergedHeaders(source), | ||
| ...conditionalHeaders | ||
| ...conditionalHeaders, | ||
| ...retryHeaders |
There was a problem hiding this comment.
BRIALERT_PROXY_URL/BRIALERT_PROXY_LIST are wired into config.mjs, but fetchText still performs plain fetch() calls without any proxy dispatcher/agent, so the proxy settings have no effect for the primary ingestion path (only Playwright uses them). In environments that require egress via proxy, most sources will continue to fail even though proxy env vars are set.
Useful? React with 👍 / 👎.
Brialert's source scraping pipeline has multiple failure modes — bot detection, brittle HTML selectors, stale URLs, timeout budget exhaustion, and long quarantine windows — that cause sources to silently drop out of rotation. This PR addresses all 10 identified failure categories across the fetch, parse, schedule, and recovery layers.
Fetch resilience
parser-failure(not justbot-block)BRIALERT_PROXY_URL/BRIALERT_PROXY_LISTenv vars, applied to bothfetch()and Playwright browser contextsFallback & recovery
replacementEndpoint/fallbackEndpoint/canonicalEndpointbefore quarantining/feed,/rss,/atom.xml, etc.) and<link rel="alternate">on the same domain/news/↔/press-releases/)quarantinedflag, fixing the bug where recovered sources stayed quarantined indefinitelyParsing
NewsArticleschema extractionog:title,og:url,article:published_time)<time>, heading tags, article containers; penalizes nav/footer)Scheduling
New sources
requiresKeywordMatch: trueKey config additions
All 83 existing tests pass; lint clean.